BUILDING AN EFFICIENT, SCALABLE, AND TRAINABLE PROBABILITY-AND-RULE- BASED PART-OF-SPEECH TAGGER OF HIGH ACCURACY by

This project is aimed to build an efficient, scalable, portable, and trainable part-of-speech tagger. Using 98% of Penn Treebank-3 as the training data, it builds a raw tagger, using Bayes’ theorem, a hidden Markov model, and the Viterbi algorithm. After that, a reinforcement machine learning algorithm and contextual transformation rules were applied to increase the tagger’s accuracy. The tagger’s final accuracy on the testing data is 96.51% and its speed is about 26,000 words per second on a computer with two-gigabyte random access memory and two 3.00 GHz Pentium duo processors. The tagger’s portability and trainability are proved by the taggermaker’s success in building a new tagger out of a corpus that is annotated with the tagset different from that of Penn Treebank. INDEX WORDS: Part-of-Speech, Tagging, Markov Model, The Viterbi Algorithm, The Baysian Theorem, Machine Learning, Contextual rules, Natural Language Processing BUILDING AN EFFICIENT, SCALABLE, AND TRAINABLE PROBABILITY-AND-RULEBASED PART-OF-SPEECH TAGGER OF HIGH ACCURACY